Efficient approximate planning in continuous space Markovian Decision Problems

MDPs provide a clean and simple, yet fairly rich framework for studying various aspects of intelligence, such as planning. A well-known practical limitation of planning in MDPs is called the curse of dimensionality [1], referring to the exponential rise in the resources required to compute (even approximate) solutions to an MDP as the size of the MDP (the number of state variables) increases. For example, conventional dynamic programming (DP) algorithms, such as valueor policy-iteration scale exponentially with the size, even if they are combined with sophisticated multigrid algorithms [4]. Moreover, the curse of dimensionality is not specific to any algorithm, as shown by a result of Chow and Tsitsiklis [3]. Recently, Kearns et al. have shown that a certain on-line, tree building algorithm avoids the curse of dimensionality in discounted MDPs [9]. Recently, this result has been extended to partially observable MDPs (POMDPs) by the same authors [8]. The bounds in these two papers are independent of the size of the state space, but scale exponentially with 1 1−γ , the effective horizon-time, where γ is the discount factor of the MDP. In this paper we consider another on-line planning algorithm that will be shown to scale polynomially with the horizon-time, as well. The price of this is that we have to assume more regularity on the MDPs we consider. In particular, we will restrict ourselves to stochastic MDPs with finite action spaces and state space X = [0, 1], and, more importantly, assume that the transition probability kernel of the MDPs are subject to the Lipschitzcondition |p(x|x1, a)− p(x|x2, a)| ≤ Lp ‖x1 − x2‖1 for any states x1, x2, x′ ∈ [0, 1] and action a ∈ A. Here Lp > 0 is a given fixed number and ‖·‖1 denotes the `1 norm of vectors. Another restriction (quite common in the literature) that we will assume is the uniform boundedness of the transition probabilities (the bound shall be denoted by Kp) and of the immediate rewards (bound denoted by Kr). Further, our bounds will depend on the dimension of the state space, d.1 The idea of the considered algorithms originates in the algorithm considered by Rust [13].2 Rust studied a more restricted class of problems than considered in this paper and proved the following result. First, let us define the concept of ε-optimality in the mean. Fix an MDP with state space X . A random, real-valued function V̂ with domain X is called ε-optimal in the mean if E [∥∥∥V̂ − V ∗∥∥∥ ∞ ] ≤ ε, where V ∗ is the optimal value function underlying the selected MDP and ‖·‖∞ is the maximum-norm and the expectation is